can you elaborate on why a mobile robot cannot avoid stationary obstacles with optical flow?

how can I test this? how did you test this? for example,

"How much does optical flow correlate with LiDAR?"
Weakly. Spearman r = +0.173 — correct direction but too noisy for a calibrated

can you show me how to reproduce this number?

Testing

I evaluated the module offline against LiDAR ground truth on the unitree_office_walk dataset (51s indoor sequence). The eval script is at scripts/eval_optical_flow_gated.py — it runs every frame through the optical flow backend, filters for frames where the robot is moving forward (speed > 0.15 m/s) and not turning (|ω| < 0.15 rad/s) to isolate pure translation, then uses the LiDAR distance on those frames as ground truth to test optical flow against. It computes:

• Correlation — Spearman r = +0.173 between optical flow τ and LiDAR TTC
• Binary detection — Precision = 0.939, Recall = 0.295 at τ=3.0 (we take Ground Truth as LiDAR distance < 1.5m)
• Latency — ~2.5 ms/frame

To reproduce, first place the unitree_office_walk dataset folder under data/, then run from repo root:

Evaluation:

PYTHONPATH=.venv/lib/python3.12/site-packages/rerun_sdk \
    uv run python3 scripts/eval_optical_flow_gated.py

Visualization (opens a Rerun viewer where you can scrub the timeline and see flow arrows, τ grid, and DANGER/CLEAR labels frame-by-frame):

PYTHONPATH=.venv/lib/python3.12/site-packages/rerun_sdk \
    uv run python3 scripts/visualize_optical_flow.py

Results land in results/optical_flow_gated.json. Full methodology and numbers are in RESULTS.md.

Stationary obstacles and optical flow

A mobile robot can avoid stationary obstacles with optical flow — as long as two conditions are met:

1. Relative motion exists — the robot is moving toward the obstacle. The robot's own forward motion produces an expanding flow field in the image (pixels diverge outward from the focus of expansion), which gives a finite τ = 1/divergence. The obstacle doesn't need to move on its own.

2. The obstacle has trackable texture — FAST needs to detect keypoints on the obstacle's surface (edges, corners, texture). A featureless surface may yield too few keypoints for a reliable divergence estimate in that grid cell.

The only case where optical flow truly can't help is when there is zero relative motion between the camera and the obstacle — e.g., the robot is stationary, or moving parallel to a wall at constant distance. No pixel displacement means zero divergence and no alarm. That's the specific case where LiDAR is still needed for proximity sensing.

• you need to understand all code written and all decisions made.
• you can do visual inspection, you don't need to do fancy LIDAR corelations, should be obvious if optical flow is detecting obstacles that are visible on screen or not
• if doing fancy LIDAR corelation - make sure it's actually correct - so visualize expected obstacles on camera image (according to lidar) small timestamp differences between odom and video, the way you take into account camera lens distortions are critical (but this is not neccessary to do at all for the first pass, just if doing this, make sure it's correct, it's better to have no data then wrong data)
• do we need this grid at all? I think there should be nicer algo that gives you a gradiant of rate of divergence of keypoints across an image? it's ok if not easy etc, just asking as I assume there is such a thing
• keypoint detection algo used is heavy and sparse, you want light (easy to compute) and non-sparse keypoints. idk if this is a diff setting for your existing keypoint algo, or entirely differnet algo

Optical flow obstacle avoidance

τ (time to contact) math

We use divergence (∂u/∂x + ∂v/∂y, computed via np.gradient on the flow field) and τ = 2 / divergence as the backend math to calculate time to contact, this approach is more robust vs the foe (focus of expansion) approach (it will break with movement around foe which gives small flow vectors).

Keypoint detector

We originally conisder FAST keypoint detector, however it is design for SLAM use (which require keypoint and descripter) and relative expensive to compute, futhermore, we need flow at every image location, including the smooth surfaces (for example, wall or some big single texture obstacle) which FAST can hardly find. Uniform grid is content-independent and constant-compute. More suitable for what we want "light + non-sparse keypoints".

Alarm

We originally go for alarming on min(τ) — single noisy cells could trip DANGER. This iteration switched to connected components on the thresholded divergence map; alarm only fires on coherent blobs (~80×80 px minimum).

Pipeline

flowchart TD
    F["Frame N-1 + Frame N"] --> LK["Lucas-Kanade on uniform grid points"]
    LK --> ERR["Drop tracks with high LK matching error"]
    ERR --> UV["raw u, v per surviving point"]

    UV --> GS["Gaussian smooth on flow field (u, v)"]
    GS -- "np.gradient to get du/dx and dv/dy" --> SOB["Per-cell divergence = (du/dx + dv/dy)"]
    SOB -- "tau = 2 / divergence " --> MS["Median smooth on divergence / tau map"]

    UV -. "raw flow" .-> A1["Arrow GEOMETRY: direction and length"]
    MS -. "smoothed tau" .-> A2["Arrow COLOR: red, yellow, green by tau band"]
    A1 --> VIZ["flow_visualization: Out Image"]
    A2 --> VIZ

    MS --> THR["Threshold mask: divergence above limit"]
    THR --> CC["Connected components on mask"]
    CC --> CHK{"Largest blob area at least 15 cells?"}
    CHK -- yes --> DT["danger_signal: Out Bool = TRUE"]
    CHK -- no --> DF["danger_signal: Out Bool = FALSE"]

   

In the GIFs below:

• Arrow direction and length = the raw (u, v) per tracked point — what LK actually returned for that frame, noise included.
• Arrow color (red / yellow / green) = the τ band from the smoothed divergence map — reflects the algorithm's per-cell danger reading after Gaussian + median smoothing. Red = τ < 3 frames, yellow = τ < 6, green = safe / not approaching.
• DANGER / CLEAR label in the corner = output of the connected-components alarm on the thresholded smoothed divergence map. Alarm only if a coherent low-τ blob exceeds ~15 cells (~80×80 px). Filter out single or small noisy blob cells.

When this module is deployed, downstream DimOS consumers subscribe to:

• danger_signal: Out[Bool] — the safety signal.
• flow_data: Out[Any] — per-point (x, y, τ, u, v) array for any custom downstream consumer/application.
• flow_visualization: Out[Image] — the annotated frame.

Demo

uv run python3 scripts/run_on_video.py [video_source]

Outdoor walk that ends at a dumpster - starts CLEAR as the dumpster is still distant, fires DANGER once it comes into range, briefly drops back to CLEAR as the camera reorients across the parking lot, then fires DANGER again at near-contact when the dumpster fills most of the frame:

Indoor scene — same algorithm, walking around a workbench with multiple obstacles (robotic arm, ArUco fixtures, cup):

Module integration

OpticalFlowModule follow the standard dimos perception module practice (In[Image] for camera, Out[Bool]/Out[Any]/Out[Image] for danger / flow data / annotated vizualize img). Production deployment is via existing dimos stack (blueprint + dimos run).

Running the full framework integration on macOS need extra configuration/integration, and somehow out of scope for this issue task #1775 which is about explore optical flow's feasability to do obstacle avoidance using mono camera video feed. To run the pipeline without extra setup, scripts/run_on_video.py calls the backend directly on video stream — same backend code, no framework needed.

Notes

• Flow field is naturally noisy (LK tracking errors, single-frame jitter). Three layers of noise handling: Gaussian smooth on the flow field before differentiating, median smooth on the divergence map after, and per-point LK reconstruction-error threshold drops bad tracks before they enter divergence at all.
• Smooth uniform surfaces (the red Solo cup, a plain wood desk) sometimes don't fire even at near-contact. LK needs features to track and smooth surfaces have few.
• iPhone is 30fps; downsampled by stride=2 to ~15fps to match go2's fps.
• Rotation breaks both LK's brightness-constancy / small-displacement assumptions and the forward-translation assumption behind τ = 2 / divergence. Module exposes angular_velocity: In[Any] and supress danger signal when |ω| > 0.3 rad/s (omega_max).
• Optical flow is an assumption-heavy approach (brightness constancy, sufficient texture to track, near-translational motion) — when assumptions break, the output degrades. So the approach is best as a cheap mono-camera second-opinion / fallback signal, not a primary obstacle avoidance guidance — Go2 already has LiDAR for that. Where this approach is useful is for platforms without depth sensors. Some future upgrade thoughs: dense flow (Farneback), learned flow (RAFT), or fusion with IMU / depth.